Azure mysql postgres auto discovery config create by GavinFrazar · Pull Request #15630 · gravitational/teleport

GavinFrazar · 2022-08-17T22:19:25Z

This PR adds config creation commands only for azure regions.

Example: teleport db configure create --azure-mysql-discovery=eastus,westus --azure-mysql-discovery=centralus

Example Output:

# Teleport database agent configuration file.
# Configuration reference: https://goteleport.com/docs/database-access/reference/configuration/
#
teleport:
  nodename: mac.attlocal.net
  data_dir: /var/lib/teleport
  auth_token: /tmp/token
  auth_servers:
  - 0.0.0.0:3080
db_service:
  enabled: "yes"
  # Matchers for database resources created with "tctl create" command.
  # For more information: https://goteleport.com/docs/database-access/guides/dynamic-registration/
  resources:
  - labels:
      "*": "*"
  # Matchers for registering Azure-hosted databases.
  azure:
  # Azure MySQL databases auto-discovery.
  # For more information about Azure MySQL auto-discovery: https://goteleport.com/docs/database-access/guides/azure-postgres-mysql/
  - subscriptions: ["*"]
    resource_groups: ["*"]
    types: ["mysql"]
    # Azure regions to register databases from.
    regions:
    - eastus,westus
    - centralus
    # Azure resource tags to match when registering databases.
    tags:
      "*": "*"
# (rest of irrelevant config omitted by me)

This is a part of a larger issue to implement Azure MySQL/PostgreSQL auto-discovery: #14688

This PR branched off #15629 so github will automatically retarget the base branch to master when that PR merges.

…overy_config_create

* Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Add config create flags for azure matchers * Add config create tests for azure * Move discovery flags for azure below aws * Fixup merge

* Azure mysql postgres auto discovery configuration (#15629) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update lib/services/matchers.go Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/services/matchers.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Remove superfluous cmp option for diffing azure matcher * Rename AzureMatchers Tags to ResourceTags * Deduplicate subscription/resource groups and add tests * Remove azure matcher config fixup Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com> * Azure mysql postgres auto discovery config create (#15630) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Add config create flags for azure matchers * Add config create tests for azure * Move discovery flags for azure below aws * Fixup merge Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Azure mysql postgres auto discovery configuration (#15629) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update lib/services/matchers.go Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/services/matchers.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Remove superfluous cmp option for diffing azure matcher * Rename AzureMatchers Tags to ResourceTags * Deduplicate subscription/resource groups and add tests * Remove azure matcher config fixup Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com> * Azure mysql postgres auto discovery config create (#15630) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Add config create flags for azure matchers * Add config create tests for azure * Move discovery flags for azure below aws * Fixup merge * Add Azure resource ID to protos (#15673) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update protobuf and fix database serialization * Update azure database spec/status * Change proto to use resource id string * Fix database serialization test Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Azure mysql postgres auto discovery configuration (#15629) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update lib/services/matchers.go Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/services/matchers.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Remove superfluous cmp option for diffing azure matcher * Rename AzureMatchers Tags to ResourceTags * Deduplicate subscription/resource groups and add tests * Remove azure matcher config fixup Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com> * Azure mysql postgres auto discovery config create (#15630) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Add config create flags for azure matchers * Add config create tests for azure * Move discovery flags for azure below aws * Fixup merge * Azure API for DB discovery (#15674) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Go mod tidy to update dependencies * Add azure response error conversion * Check for azure access denied and give a helpful error message * Add azure subscriptions api * Add azure mysql/postgresql api and wrappers * Test generic db server for azure * Make server properties its own type * Convert server types manually instead of via json * Move server list method selection logic out of api client * Update azure db server tests * Fixup merge * Update comments * Update more comments and remove junk code * Move all azure api into lib/cloud/azure * Update state and version checks * Add mutex to subscription client for caching, just in case * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/subscriptions_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Rename azure subscription client and remove sub ID caching * Add reference links for azure db ports * Move indirect dep into group * Wrap all converted azure response errors * Remove unreachable panic * Godoc DBServer * Remove maxPages arg to azure client funcs * Gofmt * Spacing between copyright and package * import order Co-authored-by: Marek Smoliński <marek@goteleport.com> * Bump go.mod version to 1.18 Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com>

* Azure mysql postgres auto discovery configuration (#15629) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Update lib/services/matchers.go Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/config/fileconf.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/services/matchers.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Remove superfluous cmp option for diffing azure matcher * Rename AzureMatchers Tags to ResourceTags * Deduplicate subscription/resource groups and add tests * Remove azure matcher config fixup Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com> * Azure mysql postgres auto discovery config create (#15630) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Add config create flags for azure matchers * Add config create tests for azure * Move discovery flags for azure below aws * Fixup merge * Azure API for DB discovery (#15674) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Go mod tidy to update dependencies * Add azure response error conversion * Check for azure access denied and give a helpful error message * Add azure subscriptions api * Add azure mysql/postgresql api and wrappers * Test generic db server for azure * Make server properties its own type * Convert server types manually instead of via json * Move server list method selection logic out of api client * Update azure db server tests * Fixup merge * Update comments * Update more comments and remove junk code * Move all azure api into lib/cloud/azure * Update state and version checks * Add mutex to subscription client for caching, just in case * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/subscriptions_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Rename azure subscription client and remove sub ID caching * Add reference links for azure db ports * Move indirect dep into group * Wrap all converted azure response errors * Remove unreachable panic * Godoc DBServer * Remove maxPages arg to azure client funcs * Gofmt * Spacing between copyright and package * import order Co-authored-by: Marek Smoliński <marek@goteleport.com> * Bump go.mod version to 1.18 * Azure MySQL and PostgreSQL DB Discovery (#15745) * Add Azure auto-discovery configuration fields * Init databases if azure matchers are in config * Use AzureMatchers in db service * Use all azure subscriptions/resource groups if omitted in matcher * Add azure config tests * Go mod tidy to update dependencies * Add azure response error conversion * Check for azure access denied and give a helpful error message * Add azure subscriptions api * Add azure mysql/postgresql api and wrappers * Test generic db server for azure * Make server properties its own type * Convert server types manually instead of via json * Move server list method selection logic out of api client * Update azure db server tests * Fixup merge * Update comments * Update more comments and remove junk code * Move all azure api into lib/cloud/azure * Update state and version checks * Convert Azure DB Server into database * Add mutex to subscription client for caching, just in case * Test database conversion from azure db server * Refactor common code * Add azure matchers to watch config * Remove unused imports * Use common max pages in memorydb users * Add azure clients to cloud clients * Add azure fetchers to watcher * Skip fetcher NotFound errors * Add azure watcher tests * Fix test comment * Add wildcard region matching * Remove redundant import * Update lib/srv/db/cloud/watchers/azure.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Move db server listing logic into separate func * Refactor some db conversion code * Use local var to cache sub IDs * Rename azure subscription client * Rename azure subscription client and remove sub ID caching * Move azure cloud clients into separate embedded interface * Update watcher test for 'NotFound' handling * Embed azureClients struct * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/subscriptions_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/cloud/azure/db_server_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Rename azure subscription client and remove sub ID caching * Add reference links for azure db ports * Move indirect dep into group * Wrap all converted azure response errors * Remove unreachable panic * Godoc DBServer * Remove maxPages arg to azure client funcs * Gofmt * Spacing between copyright and package * import order * Update lib/srv/db/cloud/watchers/azure.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Fix import order * Refactor region matching * Fix usage of azure clients to remove max pages arg * Query azure subscriptions in fetcher.Get if matching wildcard subscription * Fix azure matcher construction * Test azure fetcher discovers new subscriptions * Remove unused test helper func * Set azure database resource id metadata * Log errors when fetching from azure subscription fails * Use more readable protocol in azure db description * Get azure mysql engine version from labels * Update lib/srv/db/cloud/watchers/watcher_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> * Update lib/srv/db/cloud/watchers/watcher_test.go Co-authored-by: Marek Smoliński <marek@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com> Co-authored-by: Krzysztof Skrzętnicki <krzysztof.skrzetnicki@goteleport.com> Co-authored-by: Marek Smoliński <marek@goteleport.com>

GavinFrazar added 7 commits August 17, 2022 15:03

Add Azure auto-discovery configuration fields

f28ac40

Init databases if azure matchers are in config

ccf04e7

Use AzureMatchers in db service

eb631a8

Use all azure subscriptions/resource groups if omitted in matcher

053a2f9

Add azure config tests

38e1df7

Add config create flags for azure matchers

d634a19

Add config create tests for azure

3de022d

GavinFrazar added feature-request Used for new features in Teleport, improvements to current should be #enhancements database-access Database access related issues and PRs db/postgres PostgreSQL related database access issues db/mysql MySQL related database access issues azure labels Aug 17, 2022

GavinFrazar requested review from codingllama and smallinsky August 17, 2022 22:19

GavinFrazar self-assigned this Aug 17, 2022

GavinFrazar marked this pull request as ready for review August 17, 2022 22:21

github-actions Bot requested review from gabrielcorado and jakule August 17, 2022 22:21

jakule approved these changes Aug 18, 2022

View reviewed changes

Comment thread tool/teleport/common/teleport.go Outdated

smallinsky approved these changes Aug 18, 2022

View reviewed changes

github-actions Bot removed request for codingllama and gabrielcorado August 18, 2022 11:22

Move discovery flags for azure below aws

d4a5a52

GavinFrazar mentioned this pull request Aug 18, 2022

Azure mysql postgres auto discovery configuration #15629

Merged

Base automatically changed from gavinfrazar/azure_mysql_postgres_auto_discovery_configuration to master August 19, 2022 19:48

GavinFrazar added 2 commits August 19, 2022 12:51

Merge branch 'master' into gavinfrazar/azure_mysql_postgres_auto_disc…

f72372a

…overy_config_create

Fixup merge

40181d2

GavinFrazar enabled auto-merge (squash) August 19, 2022 19:56

Merge branch 'master' into gavinfrazar/azure_mysql_postgres_auto_disc…

ff24179

…overy_config_create

GavinFrazar merged commit d6e3202 into master Aug 19, 2022

GavinFrazar deleted the gavinfrazar/azure_mysql_postgres_auto_discovery_config_create branch August 26, 2022 00:14

GavinFrazar mentioned this pull request Aug 31, 2022

[v10] Azure mysql postgres auto discovery config create #15990

Merged

GavinFrazar added the backport/branch/v10 label Sep 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Azure mysql postgres auto discovery config create#15630

Azure mysql postgres auto discovery config create#15630
GavinFrazar merged 11 commits into
masterfrom
gavinfrazar/azure_mysql_postgres_auto_discovery_config_create

GavinFrazar commented Aug 17, 2022 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

GavinFrazar commented Aug 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

GavinFrazar commented Aug 17, 2022 •

edited

Loading